Method for graphically providing continuous change of state directions to a user of medical robotic system

ABSTRACT

Continuous change of state directions are graphically provided on a display screen to assist a user in performing necessary action(s) for transitioning between operating modes in a medical robotic system or performing corrective action. A graphical representation of a target state of an element of the medical robotic system is displayed on a display screen viewable by the user. Current states of the element and indications directing the user to manipulate the element towards the target state are continuously determined and graphical representations of the continuously determined current states and indications are displayed on the display screen along with that of the target state.

FIELD OF THE INVENTION

The present invention generally relates to medical robotic systems andin particular, to a method for graphically providing continuous changeof state directions to a user of a medical robotic system.

BACKGROUND OF THE INVENTION

Medical robotic systems such as those used in performing minimallyinvasive surgical procedures offer many benefits over traditional opensurgery techniques, including less pain, shorter hospital stays, quickerreturn to normal activities, minimal scarring, reduced recovery time,and less injury to tissue. Consequently, demand for minimally invasivesurgery using such medical robotic systems is strong and growing.

Examples of medical robotic systems include the da Vinci® SurgicalSystem and the da Vinci®S™ Surgical System from Intuitive Surgical,Inc., of Sunnyvale, Calif. Each of these systems includes a surgeon'sconsole, a patient-side cart, a high performance three-dimensional(“3-D”) vision system, and Intuitive Surgical's proprietary EndoWrist®articulating instruments, which are modeled after the human wrist sothat when added to the motions of manipulators holding the surgicalinstruments, they allow at least six degrees of freedom of motion, whichis comparable to or even greater than the natural motions of opensurgery.

The da Vinci® surgeon's console has a high-resolution stereoscopic videodisplay with two progressive scan cathode ray tubes (“CRTs”). The systemoffers higher fidelity than polarization, shutter eyeglass, or othertechniques. Each eye views a separate CRT presenting the left or righteye perspective, through an objective lens and a series of mirrors. Thesurgeon sits comfortably and looks into this display throughout surgery,making it an ideal place for the surgeon to display and manipulate 3-Dintraoperative imagery.

The patient-side cart typically includes three or more robotic armassemblies with corresponding slave manipulators for holding andmanipulating medical devices such as surgical instruments (or othertools) and image capturing devices for performing and/or viewing amedical procedure at a surgical site within a patient. To manipulatethese medical devices, the surgeon's console also includes input deviceswhich may be selectively associated with the medical devices and theirrespective slave manipulators. Since the movements of the input devicesand their associated medical devices are scaled, this allows the surgeonto perform intricate medical procedures with greater ease thanconventional open surgery as an operator of the medical robotic system.Further, it may even allow the surgeon to perform medical proceduresthat are not even feasible using conventional open surgery techniques.

To perform a minimally invasive surgical procedure on a patient, one ormore incisions are first made in the patient and cannulae insertedtherein to gain access to a surgical site within the patient. Setup armssupporting the slave manipulators are then positioned so as to allow theslave manipulators to attach to respective of the cannulae. Surgicalinstruments engaged on the slave manipulators are then inserted into thecannulae and properly positioned and oriented in order to perform theprocedure. A surgeon may then manipulate input devices which are coupledto the slave manipulators and their respective surgical instrumentsthrough one or more controllers to perform the surgical procedure.

Numerous operational modes may be implemented in a medical roboticsystem and transitions between such operational modes may be governed bysophisticated state machines. When a user wants to effect a change ofmode in the system, he or she may be required to perform one or moresteps to satisfy the relevant mode-change criteria demanded by the statemachine. When the criteria are purely discrete, they may be describedsimply with a static icon or dialog box message (e.g., “Press FaultRecover to Continue”). However, in some medical robotic systems, themode-change criteria may require the user to manipulate one or moreinput devices continuously and in a certain fashion until the actionsrequired to complete the mode change are successfully performed. In suchsystems, it may be difficult for the user to understand what he or shemust do to effect the mode change if only a simple static icon ordiscrete message is provided—thus, making the system more difficult touse and harder to learn.

OBJECTS AND SUMMARY OF THE INVENTION

Accordingly, one object of one or more aspects of the present inventionis a medical robotic system that conveys to its user the necessaryactions required to be performed by the user to effect a mode change ortake corrective action in the system.

Another object of one or more aspects of the present invention is amedical robotic system that is simple to use, easy to learn, and safe tooperate.

These and additional objects are accomplished by the various aspects ofthe present invention, wherein briefly stated, one aspect is a methodfor graphically providing continuous change of state directions to auser of a medical robotic system, the method comprising: displaying agraphical representation of a target state of an element of the medicalrobotic system on a display screen viewable by the user; continuouslydetermining current states of the element and indications directing theuser to manipulate the element towards the target state; and displayinggraphical representations of the current states of the element and theindications along with the target state on the display screen.

Another aspect is a medical robotic system comprising: an element; adisplay screen; and a processor programmed to graphically providecontinuous change of state directions to a user of the medical roboticsystem by displaying a graphical representation of a target state of theelement on the display screen, continuously determining current statesof the element along with indications directing the user to manipulatethe element towards the target state, and displaying graphicalrepresentations of the current states of the element and the indicationsalong with the target state on the display screen.

Additional objects, features and advantages of the various aspects ofthe present invention will become apparent from the followingdescription of its preferred embodiment, which description should betaken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a top view of an operating room employing a medicalrobotic system utilizing aspects of the present invention.

FIG. 2 illustrates a perspective view of an end effector of a medicaldevice used in a medical robotic system utilizing aspects of the presentinvention.

FIG. 3 illustrates a perspective view of parts of an input device usedin a medical robotic system utilizing aspects of the present invention.

FIG. 4 illustrates a block diagram of components for controlling andselectively associating device manipulators to left and righthand-manipulatable input devices in a medical robotic system utilizingaspects of the present invention.

FIG. 5 illustrates a flow diagram of a method for graphically providingcontinuous change of state directions to a user of a medical roboticsystem, utilizing aspects of the present invention.

FIGS. 6-11 illustrate various examples of graphical representationsdisplayed on a display screen for providing continuous change of statedirections to a user of a medical robotic system utilizing aspects ofthe present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 illustrates, as an example, a top view of an operating roomemploying a medical robotic system. The medical robotic system in thiscase is a minimally invasive robotic surgical system 100 including aConsole (“C”) utilized by a Surgeon (“S”) while performing a medicalprocedure, such as a diagnostic or surgical procedure, with assistancefrom one or more Assistants (“A”), on a Patient (“P”) who is recliningon an Operating table (“O”).

The Console includes a 3-D monitor 104 for displaying a 3-D image of asurgical site to the Surgeon, left and right manipulatable input devices108, 109, a foot pedal 105, and a processor 102. The input devices 108,109 may include any one or more of a variety of input devices such asjoysticks, gloves, trigger-guns, hand-operated controllers, or the like.The processor 102 may be a dedicated computer integrated into theConsole or positioned next or near to it, or it may be broken up into anumber of processing or controller components that are distributed in adistributed processing fashion throughout the system 100.

The Surgeon performs a medical procedure by manipulating the inputdevices 108, 109 (also referred to herein as “master manipulators”) sothat the processor 102 causes slave manipulators of their respectivelyassociated robotic arm assemblies 128, 129 to manipulate theirrespective removably coupled surgical instruments 138, 139 (alsoreferred to herein as “tools”) accordingly, while the Surgeon views thesurgical site in 3-D on the Console monitor 104 as it is captured by astereoscopic endoscope 140. To properly position the endoscope 140, oneor both of the input devices 108, 109 may be temporarily associated withthe endoscope 140 (as described in reference to FIG. 4) so that when theSurgeon manipulates the input device(s), the processor 102 causes aslave manipulator of the robotic arm assembly 122 to move the endoscope140 accordingly. The robotic arm assemblies 128, 129, 122 are mounted ona mobile Patient side cart 120 and communicate through electronics onthe cart 120 with the processor 102 through bus 110.

Each of the tools 138, 139, as well as the Endoscope 140, isconventionally inserted through a tool guide or cannula (not shown) intothe Patient so as to extend down to the surgical site through acorresponding minimally invasive incision such as Incision 166. Thenumber of surgical tools used at one time and consequently, the numberof robotic arms being used in the system 100 will generally depend onthe medical procedure being performed and the space constraints withinthe operating room, among other factors. If it is necessary to change atool being used during a procedure, the Assistant may remove the tool nolonger being used from its robotic arm assembly, and replace it withanother tool 131 from a Tray (“T”) in the operating room. A guided toolchange procedure may then be performed as described, for example, inU.S. Pat. No. 6,645,196 entitled “Guided Tool Change,” which isincorporated herein by reference.

Each of the robotic arm assemblies 122, 128, 129 includes a slavemanipulator and setup arms. The slave manipulators are robotically movedusing motor controlled joints (also referred to herein as “activejoints”) in order to manipulate and/or move their respectively heldmedical devices. The setup arms may be manually manipulated by releasingnormally braked joints (also referred to herein as “setup joints”) tohorizontally and vertically position the robotic arm assemblies 122,128, 129 so that their respective medical devices may be inserted intotheir respective tool guides.

Preferably, the monitor 104 is positioned near the Surgeon's hands sothat it will display a projected image that is oriented so that theSurgeon feels that he or she is actually looking directly down onto theoperating site. To that end, images of the tools 138, 139 preferablyappear to be located substantially where the Surgeon's hands arelocated.

The processor 102 performs various functions in the system 100. Oneimportant function that it performs is to translate and transfer themechanical motion of input devices 108, 109 to their respective slavemanipulators of robotic arm assemblies 128, 129 (in tool following mode)by generating and transmitting digital control signals over bus 110 sothat the Surgeon can effectively manipulate their respective tools 138,139. Another important function is to implement various controllers suchas those described in reference to FIG. 4 and the method described inreference to FIG. 5.

Although described as a processor, it is to be appreciated that theprocessor 102 may be implemented in practice by any combination ofhardware, software and firmware. Also, its functions as described hereinmay be performed by one unit, or divided up among different components,each of which may be implemented in turn by any combination of hardware,software and firmware, and distributed, in a distributed processingfashion, about the system 100.

For additional details on the construction and operation of medicalrobotic systems such as described herein, see, e.g., U.S. Pat. No.6,493,608 “Aspects of a Control System of a Minimally Invasive SurgicalApparatus,” and U.S. Pat. No. 6,424,885 “Camera Referenced Control in aMinimally Invasive Surgical Apparatus,” which are incorporated herein byreference.

FIG. 2 illustrates, as an example, a perspective view of a distal end ofa medical device 200, such as one of the surgical tools 138, 139, whichis mechanically coupled through an interface (not shown) to a slavemanipulator of a robotic arm assembly. The medical device 200 has ashaft 14.1 which is coupled at its proximal end to the interface andcoupled at its distal end 14.3 to a wrist-like mechanism 50. The medicaldevice 200 further has an end effector 58 which is coupled to thewrist-like mechanism 50.

The wrist-like mechanism 50 includes a wrist member 52. One end portionof the wrist member 52 is pivotally mounted in a clevis 17 by means of apivotal connection 54 so that the wrist member 52 can pivot in thedirection of arrows 56 about the pivotal connection 54. The end effector58 is pivotally mounted on an opposed end of the wrist member 52. Theend effector 58 is in the form of, e.g., a clip applier for anchoringclips during a surgical procedure. Accordingly, the end effector 58 hastwo parts 58.1, 58.2 together defining a jaw-like arrangement. It willbe appreciated that the end effector 58 can be in the form of anyrequired surgical tool having two members or fingers which pivotrelative to each other, such as scissors, pliers for use as needledrivers, or the like. Alternatively, it may include a single workingmember, e.g., a scalpel, cautery electrode, or the like. When a toolother than a clip applier is required during the surgical procedure, themedical device 200 may be simply removed from its associated robotic armassembly and replaced with another surgical tool, such as tool 131 inFIG. 1, bearing the required end effector, e.g., a scissors, or pliers,or the like.

The end effector 58 is pivotally mounted in a clevis 19, on an opposedend of the wrist member 52, by means of a pivotal connection 60. Freeends 11, 13 of the parts 58.1, 58.2 are angularly displaceable about thepivotal connection 60 toward and away from each other as indicated byarrows 62, 63. Members 58.1, 58.2 can be displaced angularly about thepivotal connection 60 to change the orientation of the end effector 58as a whole, relative to the wrist member 52. Thus, each part 58.1, 58.2is angularly displaceable about the pivotal connection 60 independentlyof the other, so that the end effector 58, as a whole, is angularlydisplaceable about the pivotal connection 60 as indicated in dashedlines in FIG. 2. Furthermore, the shaft 14.1 is rotatable about itscentral axis 14.2 as indicated by the arrows 59. Thus, the end effector58 has three orientational degrees of freedom of movement relative tothe working end 14.3, namely, rotation about the axis 14.2 as indicatedby arrows 59, angular displacement as a whole about the pivot 60 andangular displacement about the pivot 54 as indicated by arrows 56. Theorientational movement of the end effector 58 is controlled byappropriately positioned actuators (e.g., electrical motors) in itscoupled to robotic arm assembly, which respond to inputs from itsassociated input device through the processor 102 to drive the endeffector 58 to a desired orientation as dictated by movement of theinput device.

FIG. 3 illustrates, as an example, a perspective view of a hand-held orhand-grippable part 300 of the input device (e.g., 108 or 109)associated with the medical device 200. The part 300 has an articulatedarm portion including a plurality of members or links 302 connectedtogether by joints 304. The Surgeon grips the part 300 by positioninghis or her thumb and index finger over a pincher formation 306. Thejoints of the part 300 are operatively connected to electric motors toprovide for, e.g., force feedback, gravity compensation, and/or thelike. Appropriately positioned sensors, e.g., encoders, orpotentiometers, or the like, are positioned on each joint of the part300, so as to enable the joint positions to be determined for commandingmovement of the medical device 200 through a control system implementedin the processor 102 to drive its associated slave manipulator.

As shown in FIG. 3, the part 300 has a plurality of degrees of freedommovement. First of all, the part 300 is mountable on another part of theinput device that facilitates translational movement of the part 300.Secondly, the pincher formation 306 of the part 300 is rotatable aboutseveral axes shown as dotted lines in FIG. 3. In particular, the pincherformation 306 may be rotated in a direction 1 so as to command the endeffector 58 to rotate about the axis 14.2 as indicated by arrows 59, thepincher formation 306 may be rotated in a direction 2 to command the endeffector 58 to rotate about the pivot 60, and the pincher formation 306may be rotated in a direction 3 to command the end effector 58 to rotateabout the pivot 54 as indicated by arrows 56.

Further, when the pincher formation 306 of the hand-grippable part 300is squeezed between the thumb and index finger of the Surgeon, thefingers 58.1, 58.2 of the end effector 58 close. When the thumb andindex finger of the Surgeon cause the pincher formation 306 to be movedapart, the fingers 58.1, 58.2 of the end effector 58 move apart insympathy.

FIG. 4 illustrates, as an example, a block diagram of components forcontrolling and selectively associating device manipulators to the inputdevices 108, 109. Various surgical tools such as graspers, cutters, andneedles may be used to perform a medical procedure at a work site withinthe Patient. In this example, two surgical tools 138, 139 are used torobotically perform the procedure and the camera 140 is used to view theprocedure. The tools 138, 139 and camera 140 are inserted through theirrespective ports into the Patient using the setup portion of theirrespective robotic arm assemblies 128, 129, 122 and maneuvered by slavemanipulators of their respective robotic arm assemblies 128, 129, 122towards the work site where the medical procedure is to be performed.

Each of the devices 138, 139, 140 is manipulated by its own slavemanipulator. In particular, the camera 140 is manipulated by a cameramanipulator (ECM) 412, the first surgical tool 139 is manipulated by afirst tool manipulator (PSM1) 432, and the second surgical tool 138 ismanipulated by a second tool manipulator (PSM2) 442. So as to not overlyencumber the figure, the devices 138, 139, 140 are not shown, only theirrespective manipulators 442, 432, 412 are shown in the figure.

In this example, each of the input devices 108, 109 may be selectivelyassociated with one of the devices 138, 139, 140 so that the associateddevice may be controlled by the input device through its controller andmanipulator. For example, by placing switches 458, 459 respectively intool following modes “T2” and “T1”, the left and right input devices108, 109 may be respectively associated with the second and firstsurgical tools 138, 139, which are telerobotically controlled throughtheir respective controllers 443, 433 and manipulators 442, 432 so thatthe Surgeon may perform a medical procedure on the Patient while thecamera 140 is soft-locked in place by its controller 413.

When the camera 140 is to be repositioned by the Surgeon, either one orboth of the left and right input devices 108, 109 may be associated withthe camera 140 so that the Surgeon may move the camera 140 through itscontroller 413 and manipulator 412. In this case, the disassociatedone(s) of the surgical tools 138, 139 is soft-locked in place by itscontroller. For example, by placing switches 458, 459 respectively incamera positioning modes “C2” and “C1”, the left and right input devices108, 109 may be associated with the camera 140, which is teleroboticallycontrolled through its controller 413 and manipulator 412 so that theSurgeon may position the camera 140 while the surgical tools 138, 139are soft-locked in place. If only one input device is to be used forpositioning the camera, then only one of the switches 458, 459 is placedin its camera positioning mode while the other one of the switches 458,459 remains in its tool following mode so that its respective inputdevice may continue to control its associated surgical tool.

The selective association of the input devices 108, 109 to other devicesin this example may be performed by the Surgeon using a Graphical UserInterface (GUI) or a voice recognition system implemented on the SurgeonConsole. Alternatively, the association of the input devices 108, 109may be changed by the Surgeon depressing a button on one of the inputdevices 108, 109, depressing the foot pedal 105, or using any other wellknown mode switching technique.

After the Surgeon has temporarily associated the input devices 108, 109with the camera 140 and moved the camera 140 to its desired positionand/or orientation to gain a better view of a worksite, the Surgeon mustthen re-associate the input devices 108, 109 with their formerlyassociated surgical tools 138, 139 before performing a medical procedureat the worksite. In order to avoid abrupt movement of the surgical tools138, 139 due to their respective controllers 443, 433 correcting fortheir misalignment with the input devices 108, 109, it is desirable tore-align the input devices 108, 109 with the surgical tools 138, 139before effecting the re-association. The steps necessary to re-align theinput devices 108, 109 with the soft-locked surgical tools 138, 139 maynot be readily known by the Surgeon, however. Therefore, it would bedesirable for the medical robotic system 100 to provide guidance to theSurgeon to perform the re-alignment.

FIG. 5 illustrates a flow diagram of a method for graphically providingcontinuous change of state directions to a user of the medical roboticsystem 100 and FIGS. 6-11 illustrate several examples of graphicalrepresentations generated and displayed by the method on a displayscreen of the monitor 104 of the medical robotic system.

In 501, an operational mode change is initiated. One example of such amode change is the above-described re-alignment of the input device witha surgical tool prior to re-association of the input device with thesurgical tool. Examples of this type of mode change are shown in FIGS.6-8. Another type of mode change is initiated when a joint of a slavemanipulator reaches or becomes dangerously close to a limitation in itsrange of motion. In this case, control of the slave manipulator may betemporarily suspended (and/or a feedback force may be exerted againstthe input device) until the user moves the input device to a state thatno longer commands exceeding the range of motion joint limit. An exampleof this type of mode change is shown in FIG. 9. Yet another type of modechange is initiated when a medical device is first inserted through aguide tube into a Patient. In this case, the mode change is completedwhen the end effector of the medical device exits the distal end of theguide tube. An example of this type of mode change is shown in FIG. 10.Still another type of mode change is a guided tool change in which areplacement tool is guided to a position formerly held by the tool whichit has replaced. An example of this type of mode change is shown in FIG.11. It is to be appreciated that the mode changes described herein aremerely examples and that many other types of mode changes (andcorrective actions within a mode) may also be used with the methoddescribed in reference to FIG. 5 and therefore, are contemplated asbeing within the full scope of various aspects of the present invention.

In 502, a target state for an element of the medical robotic system 100is determined. The element can be any manipulatable component of themedical robotic system that is continuously manipulated by the user toperform the requisite steps for a mode change. For example, the elementmay be the input device and more particularly, a pincher formation ofthe input device or, as another example, the element may be the medicaldevice and more particularly, an end effector of the medical device.Note that the graphical representation of the element does notnecessarily have to be a depiction of the element.

The target state is generally the state of the element that is requiredto satisfy the mode change criteria. For example, in FIG. 6, the targetstate of the element (i.e., pincher formation 306) is a pincher angleindicated by stop gap 601 which corresponds to the angle of the openingof the end effector jaws of the medical device, in its soft-lockedstate, prior to re-association of the input device with the medicaldevice. Note that in this example and others described herein, thegraphical representation of the pincher formation 306 is identified as306′ to distinguish it from the actual pincher formation 306. As anotherexample, in FIG. 8, the target state of the element (i.e., pincherformation 306) is a roll angle indicated by stop gap 801 whichcorresponds to the roll angle of the end effector of the medical device,in its soft-locked state, prior to re-association of the input devicewith the medical device. As another example, in FIG. 9, the target stateof the element (e.g., pincher formation 306) is a position 901 which issafely away from an initial position 910 at which one or more joints ofthe slave manipulator of the medical device has reached its range ofmotion limit or come within a threshold distance away from its range ofmotion limit. As another example, in FIG. 10, the target state of theelement (e.g., end effector 58) is an extended position of the medicaldevice indicated by stop gap 1001 at which its end effector exits thedistal end of a guide tube (e.g., a cannula) 1010 in which the medicaldevice has been inserted for entry into the Patient. Note that in thisexample and others described herein, the graphical representation of theend effector 58 is identified as 58′ to distinguish it from the actualend effector 58. As another example, in FIG. 11, the target state of theelement (e.g., end effector 58) is the position and orientation that thereplacement tool is to be moved to in order to occupy the same statethat the tool which it is replacing previously occupied for performing amedical procedure at a worksite within a Patient. The target state inthis example is graphically represented advantageously by athree-dimensional ghost image 1101 of the end effector so that bothposition and orientation may be indicated.

In 503, a graphical representation of the determined target state isdisplayed on the display screen 104. Typically, the graphicalrepresentation of the target state depicts the element such as shown inFIGS. 6, 8-11. However, this is not a necessary requirement of themethod of FIG. 5. For example, in both FIGS. 6, 7, the same mode changeis being performed in which the element is the input device and the modechange involves re-aligning the open angle of the pincher formation withthe open angle of the end effector's jaws before re-associating themedical device with the input device. In FIG. 6, the graphicalrepresentation of the target state of the element depicts the pincherformation 306. However, in FIG. 7, it depicts the state of the endeffector 58 which would be commanded by the pincher formation 306 if themedical device were associated with the input device at the time. Thus,the target state graphically depicted by stop bar 701 in FIG. 7indicates a command corresponding to the held position of the endeffector 58.

In 504, a current state of the element is determined. When the elementis the input device (or part thereof), the determination of its currentstate is preferably performed using sensed joint positions of the inputdevice and applying the sensed positions to forward kinematics of theinput device. On the other hand, when the element is the medical device(or part thereof), the determination of its current state is preferablyperformed using sensed joint positions of the slave manipulatormanipulating the medical device and applying the sensed positions toforward kinematics of the slave manipulator.

In 505, a determination is made whether the current state matches thetarget state (i.e., the current state of the element has reached thetarget state of the element).

To perform this comparison, it is useful to first determine the targetstate of the element in same coordinate system as the current state ofthe element. Although described at this point, such determination may beadvantageously performed in 502 above. In the case where the targetstate is a held (soft-locked) position of the medical device, such as inthe examples of FIGS. 6-8, the held state may be determined using sensedjoint positions of the slave manipulator manipulating the medical deviceand applying the sensed positions to forward kinematics of the slavemanipulator. In the case where the target state is determined by backingoff to a safe distance from a current joint position of the slavemanipulator, such as in the example of FIG. 9, the target state may bedetermined by first determining the current position of the device andbacking off from that position by the safe distance. The currentposition of the device in this case may be determined using sensed jointpositions of the slave manipulator manipulating the device and applyingthe sensed positions to forward kinematics of the slave manipulator. Inthe case where the target state depends on the position of anotherstructure, such as in the guide tube example of FIG. 10, the position ofthe guide tube may be determined using sensed joint positions of a slavemanipulator manipulating the guide tube and applying the sensedpositions to forward kinematics of the slave manipulator. In the casewhere the target state is the former position of a replaced medicaldevice, such as in the example of FIG. 11, the position of the medicaldevice that is being replaced may be determined using sensed jointpositions of the slave manipulator manipulating the medical device andapplying the sensed positions to forward kinematics of the slavemanipulator. The determined position is then stored in a non-volatilememory of the medical robotic system so that it may be used forgenerating the target state at a later time.

Now if the determination in 505 is YES, then in 508, the operationalmode change is completed since the element has successfully reached itstarget state. As an example of such completion, with respect to FIGS.6-8, the medical device is re-associated with the input device. Asanother example of such completion, with respect to FIG. 9, the roboticcontrol of the medical device is re-activated since its slavemanipulator is no longer in danger of reaching a range of motion limiton one of its joints.

On the other hand, if the determination in 505 is NO, then in 506, oneor more indicators are determined that would direct the user to move theelement towards its target state. Determination of what the indicator(s)should indicate in this case is straightforward since the current andtarget states are known. It is important to note, however, that theindicator(s) is/are not static. The indicator(s) may continually changeas the element is manipulated by the user towards the target state.Thus, the indicator(s) is/are dynamic in this sense.

In 507, graphical representations of the current state and the one ormore indicators are displayed on the display screen 104 along with thegraphical representation of the target state. As examples, the currentstate of the pincher formation angle is indicated by arc line 602 andits corresponding indicator by arrow 603 in FIG. 6; the current state ofthe pincher formation angle is indicated by the commanded jaw position702 and its corresponding indicator by arrow 703 in FIG. 7; the currentstate of the pincher formation roll angle is indicated by point 802 andits corresponding indicator by arrow 803 in FIG. 8; the current state ofthe pincher formation is indicated by a graphical representation 902 ofthe pincher formation and its corresponding indicator by arrow 903 inFIG. 9; the current state of the end effector is indicated by a line1002 shown in front of a graphical representation of the end effectorand its corresponding indicator by arrow 1003 in FIG. 10; and thecurrent state of the end effector is indicated by a graphicalrepresentation 1102 of the end effector and its corresponding indicatorby arrow 1103 in FIG. 11. Although each of these examples shows agraphical representation for only a single indicator, more than oneindicator may be depicted at one time. For example, the pincher openingand roll angle targets of FIG. 6, 8 may be combined into a singledisplay along with their respective current states 602, 802 andindicators 603, 803. Thus, for each degree of freedom movement, aseparate arrow indicator may be provided to indicate required movementin that degree of freedom.

The method then loops back to 503 to process sampled data values readfrom position sensors associated with the element for a next processcycle through 503-507 and continues looping through 503-507 until a YESdetermination is made in 505 and the method is completed in 508. Thus,change of state directions are graphically provided to the user on acontinuous basis until the target state is reached.

Although the various aspects of the present invention have beendescribed with respect to a preferred embodiment, it will be understoodthat the invention is entitled to full protection within the full scopeof the appended claims. In particular, although the method of FIG. 5describes its use for performing a mode change, it is to be appreciatedthat inventive aspects of the method may also be used within a modewhere dynamic change of state actions are required by the user.

1-29. (canceled)
 30. A method implemented by a processor for graphicallyproviding continuous change of state directions to a user of a medicalrobotic system during a tool exchange procedure in which an old tool isexchanged with a replacement tool, the method comprising: displaying agraphical representation of at least one target state for thereplacement tool on a display screen viewable by the user, wherein theat least one target state comprises one or both of a stop gap indicatinga position at which an end effector of the replacement tool exits adistal end of a guide tube as part of the tool exchange procedure and anoperational position and orientation of an end effector of the old toolprior to retraction of the old tool into the guide tube as part of thetool exchange procedure; continuously determining current states of theend effector of the replacement tool and indications directing the userto manipulate the end effector of the replacement tool towards the atleast one target state starting from an initial position of the endeffector of the replacement tool within the guide tube; and continuouslydisplaying graphical representations of the current states of the endeffector of the replacement tool and graphical representations of theindications on the display screen along with the graphicalrepresentation of the at least one target state.
 31. The method of claim30, wherein the graphical representation of the at least one targetstate comprises a line which is indicative of the stop gap and isorthogonal to an insertion direction indicated by a path of thedetermined current states of the end effector of the replacement tool asit moves from the initial position towards the target state.
 32. Themethod of claim 30, wherein the graphical representation of the at leastone target state comprises a ghost image of the end effector of thereplacement tool, wherein the ghost image is disposed at the operationalposition and orientation of the end effector of the old tool prior toretraction of the old tool into the guide tube as part of the toolexchange procedure.
 33. The method of claim 30, wherein the graphicalrepresentation of the current state of the end effector of thereplacement tool comprises a graphical representation of the endeffector of the replacement tool that is disposed on the display screenso as to indicate a current position and orientation of the end effectorof the replacement tool relative to the target state.
 34. The method ofclaim 30, wherein the graphical representation of the indicationcomprises at least one arrow directing the user to manipulate the endeffector of the replacement tool towards the at least one target. 35.The method of claim 34, wherein the at least one arrow comprises anarrow pointing from the graphical representation of the current state ofthe end effector of the replacement tool towards the line which isindicative of the stop gap.
 36. The method of claim 34, wherein the atleast one arrow comprises an arrow pointing from the graphicalrepresentation of the current state of the end effector of thereplacement tool towards the ghost image of the end effector of thereplacement tool.
 37. A medical robotic system comprising: an old toolhaving an end effector; a replacement tool having an end effector; aguide tube; a slave manipulator adapted to removably hold and manipulatethe old tool and the replacement tool; a display screen viewable by auser; and a processor programmed to: cause displaying of a graphicalrepresentation of at least one target state for the replacement tool onthe display screen, wherein the at least one target state comprises oneor both of a stop gap indicating a position at which the end effector ofthe replacement tool exits a distal end of the guide tube when thereplacement tool is being held and manipulated by the slave manipulatoras part of a tool exchange procedure and an operational position andorientation of the end effector of the old tool prior to retraction ofthe old tool into the guide tube when the old tool is being held andmanipulated by the slave manipulator as part of the tool exchangeprocedure; continuously determine current states of the end effector ofthe replacement tool and indications directing the user to manipulatethe end effector of the replacement tool towards the at least one targetstate starting from an initial position of the end effector of thereplacement tool within the guide tube when the replacement tool isbeing held and manipulated by the slave manipulator as part of the toolexchange procedure; and cause continuous displaying of graphicalrepresentations of the current states of the end effector of thereplacement tool and graphical representations of the indications on thedisplay screen along with the graphical representation of the at leastone target state when the replacement tool is being held and manipulatedby the slave manipulator as part of the tool exchange procedure.
 38. Themedical robotic system of claim 37, wherein the graphical representationof the at least one target state comprises a line which is indicative ofthe stop gap and is orthogonal to an insertion direction indicated by apath of the determined current states of the end effector of thereplacement tool as it moves from the initial position towards thetarget state.
 39. The medical robotic system of claim 37, wherein thegraphical representation of the at least one target state comprises aghost image of the end effector of the replacement tool, wherein theghost image is disposed at the operational position and orientation ofthe end effector of the old tool prior to retraction of the old toolinto the guide tube as part of the tool exchange procedure.
 40. Themedical robotic system of claim 37, wherein the graphical representationof the current state of the end effector of the replacement toolcomprises a graphical representation of the end effector of thereplacement tool that is disposed on the display screen so as toindicate a current position and orientation of the end effector of thereplacement tool relative to the target state.
 41. The medical roboticsystem of claim 37, wherein the graphical representation of theindication comprises at least one arrow directing the user to manipulatethe end effector of the replacement tool towards the at least onetarget.
 42. The medical robotic system of claim 41, wherein the at leastone arrow comprises an arrow pointing from the graphical representationof the current state of the end effector of the replacement tool towardsthe line which is indicative of the stop gap.
 43. The medical roboticsystem of claim 41, wherein the at least one arrow comprises an arrowpointing from the graphical representation of the current state of theend effector of the replacement tool towards the ghost image of the endeffector of the replacement tool.